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Abstract 

Naval  vessels,  like  most  large-capital  projects,  have  a  long  history  of  cost  growth.  To 
get  a  handle  on  this  problem,  NAVSEA’s  Cost  Engineering  &  Industrial  Division,  NAVSEA  05C, 
has  introduced  Probabilistic  Cost  Risk  Analysis  (PCRA)  into  the  Department  of  Defense  (DoD) 
Planning,  Programmatic,  Budgeting,  and  Execution  System  (PPBES).  The  quantification  of  cost 
in  terms  of  cumulative  probability  distribution  functions  (CDF)  or  “S-curves”  provides  a 
macroscopic  view  of  project  risk.  Risk  curves  alone  do  not  provide  adequate  visibility  into  the 
individual  project  risk  drivers;  therefore,  they  are  insufficient  for  planning  and  managing  risk 
reduction  activities  (RRA).  Complex  projects  typically  involve  a  set  of  high-consequence, 
project-specific  risks  that  require  detailed  analysis  and  for  which  risk  response  actions  need  to 
be  developed  and  implemented.  The  analysis  of  specific  risks  and  RRAs  requires  a  microscopic 
view.  We  present  a  practical  and  mathematically  sound  approach  using  scenarios  and  Monte 
Carlo  simulation  within  the  framework  of  decision  trees  and  risk  curves.  The  approach  is 
detailed  using  a  realistic  but  simplified  case  of  a  project  with  three  technical  risks. 
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Introduction 


Cost  growth  has  been  a  major  problem  for  the  US  Navy.  Over  the  past  four  decades, 
the  growth  of  US  Navy  ship  costs  has  exceeded  the  rate  of  inflation.  In  the  past  50  years, 
annual  cost  escalation  rates  for  amphibious  ships,  surface  combatants,  attack  submarines,  and 
nuclear  aircraft  carriers  have  ranged  from  7  to  11  %  (Arena,  Blickstein,  Younossi  &  Grammich, 
2006).  Along  with  real  cost  growth,  the  DoD  has  had  significant  problems  with  cost  estimates. 
By  and  large,  the  DoD  and  the  military  departments  have  underestimated  the  cost  of  buying 
new  weapon  systems.  A  recent  study  by  RAND  (Arena,  Leonard,  Murray  &  Younossi,  2006) 
indicates  that  there  is  a  systematic  bias  toward  underestimating  weapon  system  costs  and 
substantial  uncertainty  in  estimating  the  final  cost  of  a  weapons  system. 

The  DoD  recognizes  that  uncertainty  is  an  important  part  of  cost  estimating.  During  a 
2007  seminar  with  a  naval  aviation  program  official,  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition,  Dr.  Etter  (Burgess,  2007),  stated: 


Program  managers  not  only  need  to  know  a  realistic  cost  estimate  for  their  program, 
they  need  to  know  the  percent  probability  of  achieving  that  target.  For  example,  a  ship 
with  a  40%  chance  of  coming  in  on  budget  has  a  60%  chance  of  being  over  budget. 

Such  a  situation  should  prompt  the  project  manager  to  seek  help  from  the  acquisition 
community,  (p.  42) 

There  is  an  ongoing  major  shift  in  R&D  and  complex  engineering  projects  from 
deterministic  to  probabilistic  approaches.  Probabilistic  Cost  Analysis  (PCA)  provides  the  proper 
framework  for  handling  the  many  different  elements  of  cost  uncertainty,  including  project- 
specific,  high-consequence  risks.  These  risk  drivers  must  be  identified,  assessed,  mitigated, 
and  controlled  through  formal  risk  management — which  is  an  essential  and  critical  discipline 
implemented  in  today’s  DoD  projects.  The  Risk  Management  Guide  for  DoD  Acquisition  (2006) 
reads: 


Risk  management  is  a  continuous  process  that  is  accomplished  throughout  the  life  cycle 
of  a  system.  It  is  an  organized  methodology  for  continuously  identifying  and  measuring 
the  unknowns;  developing  mitigation  options;  selecting,  planning,  and  implementing 
appropriate  risk  mitigations;  and  tracking  the  implementation  to  ensure  successful  risk 
reduction.  Effective  risk  management  depends  on  risk  management  planning;  early 
identification  and  analyses  of  risks;  early  implementation  of  corrective  actions; 
continuous  monitoring  and  reassessment;  and  communication,  documentation,  and 
coordination,  (p.  3) 

The  DoD  considers  the  definition,  implementation  and  documentation  of  risk 
management  essential  to  acquisition  success.  The  DoD  risk  management  process  outlined  in 
the  Risk  Management  Guide  consists  of  the  following  five  activities  performed  on  a  continuous 
basis:  Risk  Identification,  Risk  Analysis,  Risk  Mitigation  Planning,  Risk  Mitigation 
Implementation,  and  Risk  Tracking. 

This  process  is  consistent  with  the  AACE  definition,  which  includes  identifying  and 
analyzing  risk  factors  or  drivers,  mitigating  the  risk  drivers  where  appropriate,  estimating  their 
impact  on  plans  and  monitoring  and  controlling  risk  during  execution  (Hollman,  2006).  To  be 
effective,  PCA  must  interface  with  each  of  the  risk  management  activities. 
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An  emphasis  on  risk  management  supports  efforts  to  reduce  lifecycle  costs  of  system 
acquisitions.  An  often-neglected  concept  in  project  risk  management  is  the  consideration  of  the 
entire  project  lifecycle.  Analysis  of  risk  over  the  lifecycle  of  a  system  can  yield  substantial 
benefits.  Conversely,  ignoring  important  stages  of  the  lifecycle  can  lead  to  substantial  problems 
in  terms  of  risk  for  product  development  at  the  beginning  of  the  lifecycle  and  for  product  upgrade 
or  replacement  at  the  end  (Pennock  &  Haimes,  2001 ). 

Many  sources  of  cost  uncertainty  in  naval  vessel  construction — such  as 
economic/business  factors  (rates-wages,  overhead,  G&A,  vendor/supplier  stability,  inflation 
indices,  multi-year  assumptions,  etc.),  learning/rate/curve  assumptions,  and  cost-reduction 
initiatives — are  well  understood  within  the  framework  of  a  macroscopic  perspective;  these  are 
effectively  modeled  with  classical  Probability  Distribution  Functions  (PDF)  such  as  the 
triangular.  Beta,  lognormal,  and  Weibull  distributions.  However,  these  factors  constitute  only  a 
fraction  of  today’s  typical  project  risk  drivers  and,  therefore,  cost  uncertainty. 

The  construction  of  naval  vessels,  like  most  complex  engineering  projects,  is  also 
susceptible  to  project-specific  risk  drivers,  such  as:  low  Technology  Readiness  Level  (TRL); 
high  design,  manufacturing,  and  complexity;  significant  requirement  changes;  sizeable  quantity 
changes;  large  funding  uncertainty;  severe  acts  of  nature;  and  serious  accidents. 

It  is  tempting  to  assume  or  claim  that  the  PDFs  typically  elicited  for  cost  elements  also 
quantify  the  project-specific,  high-consequence  risks.  Sometimes  cost  analysts  will  go  through 
the  effort  of  identifying  and  discussing  risk  drivers,  but  when  it  comes  to  quantifying  the  risks 
and  estimating  contingency,  they  simply  apply  high/low  ranges  to  WBS  elements  without 
thinking  about  how  a  particular  risk  driver  affects  one  or  more  cost  elements.  We  think  it  is 
invalid  and  counterproductive  to  do  this  because  it  leads  to  the  loss  of  valuable  information  and 
visibility  into  these  risks.  Also,  this  approach  tends  to  focus  on  cost  reduction  rather  than  risk 
mitigation.  Hollmann  (2007)  notes  that  in  best  practice,  risk  analysis  should  begin  with  the 
identification  of  risk  drivers  and  events.  The  cost  impacts  of  the  risk  drivers  and  events  are  then 
considered  specifically  for  each  event. 

The  analysis  of  specific  risks  and  Risk  Response  Actions  (RRA)  requires  a  microscopic 
view  and  is  best  carried  out  with  tools  such  Decision  Trees  (DT),  influence  diagrams,  or  other 
discrete  representations.  This  microscopic  perspective  offers  many  benefits.  It  is  a  powerful 
risk  analysis  method  to  explicitly  model  high-consequence  risks  and  RRAs,  and  thereby 
provides  a  tool  for  making  better  decisions.  It  also  assists  subject-matter  experts  (SMEs)  to 
think  about  credible,  high-consequence  events  and  better  deal  with  overconfidence  or  optimism 
biases.  However,  the  microscopic  view  is  too  cumbersome  to  individually  analyze  every  risk 
and  source  of  cost  uncertainty.  It  complements  and  needs  to  be  integrated  within  the  PCA. 

In  this  paper,  we  propose  to  develop  a  microscopic/macroscopic  PCA  as  an  integral 
entity  of  the  DoD  risk  management  process,  as  follows: 

1 .  The  cost  and/or  risk  analyst  (simply  referred  to  as  analyst  below)  and  the  SMEs  jointly 

identify  the  individual  risks  using  the  standard  DoD  risk-identification  process. 

2.  The  analyst  and  the  SMEs  jointly  screen  the  identified  risks  for  further  analysis  and  risk 

mitigation. 

3.  The  analyst  and  SMEs  jointly  identify  realistic  RRAs  for  the  screened  risks. 
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4.  The  analyst  models  each  risk  and  its  RRAs  using  a  DT. 

5.  The  analyst  works  with  the  SMEs  to  quantify  the  value  of  the  decisions  and  outcomes  for 
each  DT  using  discrete  and  continuous  distributions.  We  favor  the  Direct  Fractile 
Assessment  (DFA)  method  for  data  elicitation  and  fitting  the  associated  cost  elements 
with  a  three-parameter  Weibull  distribution. 

6.  The  analyst  quantifies  the  DTs  using  Monte  Carlo  simulation.  Risks  and  RRAs  are  then 
modeled  in  terms  of  risk  curves.  We,  thereby,  avoid  relying  on  the  minimum  expected 
risk  value,  which  is  a  serious  shortcoming  of  standard  decision  analysis. 

7.  The  analysis  is  readily  performed  using  commercial  Excel  add-ins  (Crystal  Ball, 

@Risk...)  or  more  specialized  tools  (DecisionPro,  Analytica...). 

The  goal  of  this  paper  is  to  present  a  realistic  and  practical  method  for  explicitly 
analyzing  and  controlling  the  cost  impact  of  project  risks  and  realistic  RRAs.  Projects  can  then 
dynamically  determine  the  optimal  temporal  set  of  decision  gates  for  a  given  probability  of 
success — thereby  reducing  cost  while  increasing  the  probability  of  project  success.  We 
illustrate  the  method  using  a  realistic  but  simplified  case  of  a  project  with  three  technical  risks. 
We  close  with  some  concluding  remarks  and  recommendations  for  further  development. 

The  Quantification  of  Multiple  Project  Risks 

Consider  a  project  with  n  credible,  high-consequence  risks  {RJ.  Each  risk,  Ri,  is 
characterized  by  a  probability  of  occurrence  Pi  and  a  spectrum  of  possible  outcomes  with  a  PDF 
Li(x),  where  x  is  a  random  variable  that  represents  the  magnitude  of  the  associated  cost  or  loss. 
One  may  then  think  of  this  set  of  risks  as  a  risk  portfolio  or  repository  (Kujawski  &  Miller,  2007) 
with  a  generalized  discrete  PDF  Rs(x)  given  by: 


Equation  1.  Rs(x) 


|(Pi ,  Li  (x)) ,  (P2 ,  Lj  (x)) , . . . ,  (p„ ,  L„  (x)) , 


1-Zp.o 


i=l 


The  total  project  cost  is  a  random  variable  that  consists  of  the  sum  of  the  m  base  cost 
elements  {BCJ  and  the  explicitly  identified  risk  costs  {RCi}.  Depending  on  the  state  of 
knowledge  of  the  data,  the  base  cost  elements  BCi  may  be  modeled  as  either  point  estimates  or 
continuous  PDFs.  The  total  project  cost  TC  is  then  the  probabilistic  sum  of  the  m  base  cost 
elements  and  n  risk-driver  costs: 


Equation  2.  TC{x)  =  ^  BCi{x)  -i-^  piLi{x) 

i=l  ;=1 

Equations  1  and  2  provide  visibility  into  the  link  between  the  credible,  high-consequence 
risks  {RJ  and  the  total  project  cost-risk  curve.  Monte  Carlo  simulation  tools  such  as  Crystal  Ball 
and  @Risk  can  also  provide  tornado  charts  that  conveniently  quantify  the  importance  of  the 
various  risk  drivers  and  their  link  to  the  overall  cost  risk.  Projects  can  use  this  information  to 
rationally  identity  risks.  This  is  in  sharp  contrast  with:  (1)  the  use  of  point  estimates  that  are  at 
best  ambiguous  because  overly  confident  staff  provide  low  cost  estimates,  while  others  may 
inflate  their  cost  estimates  to  make  it  easier  to  achieve  success,  (2)  decision-making  based  on 
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qualitative  assessments,  and  (3)  the  consideration  of  only  S-curves,  which  only  provide  a 
macroscopic  and  somewhat  “black  box”  view  of  project  risk  and  cost  uncertainty. 

Modeling  and  Analyzing  Risk  Response  Actions 

We  model  and  analyze  each  screened  risk  and  the  proposed  RRAs  using  a  generalized 
DT — ^where  PDFs  rather  than  discrete  branches  are  associated  with  the  chance  nodes,  and  the 
outcomes  are  analyzed  using  Monte  Carlo  simulation  (Kujawski,  2002).  This  provides  a 
powerful  technique  for  dealing  with  the  complex  situations  typical  of  today’s  DoD  projects.  It 
avoids  bushy  trees  and  generates  risk  curves,  thereby  removing  the  reliance  of  decision-making 
based  on  expected  value. 

To  illustrate  the  approach,  consider  the  risk  depicted  in  Figure  1 .  To  be  concrete.  Risk 
#1  is  associated  with  fabricating  a  complex  module.  The  two  risk  response  actions  are:  (i) 
Directly  fabricate  the  module,  or  (ii)  Build  a  prototype  and  then  fabricate  the  module.  The 
generalized  DT  follows  the  standard  DT  representation.  Decision  nodes  and  chance  nodes  are 
depicted  as  squares  and  circles,  respectively.  The  branches  that  originate  with  decision  nodes 
represent  the  available  RRAs.  The  branches  that  originate  with  chance  nodes  represent  the 
possible  probabilistic  outcomes.  A  descriptive  label,  a  probability,  and  a  cost  distribution  are 
associated  with  each  branch.  These  probability  and  cost  values  are  conditional  on  the  RRA  and 
may  also  be  conditional  on  the  outcome  of  other  risks  in  case  of  interdependencies.  We  model 
the  cost  values  using  a  three-parameter  Weibull  distribution  fitted  to  the  10*'^,  50*'^,  and  90*'^ 
percentiles  determined  in  accordance  with  the  Direct  Fractile  Assessment  (DFA)  method. 

In  this  example,  we  assume  that  the  baseline  cost  is  $1,1  OOK.  Risk  is  then  given  by  the 
Value  At  Risk  (VAR)  relative  to  this  value.  The  VAR  corresponds  to  the  events  whereby 
production  of  the  module  exceeds  $1,1  OOK.  The  ordering  of  the  decision  nodes  corresponds  to 
different  temporal  deterministic  events  in  the  development  and  fabrication  cycle  of  the  module. 


Risk  #1 


Fob  A2 


80%  Success  A1 


Fab  A1 


Prototype 

too 


Success_AlF 
95%  VAR-(900. 1100, 14001  IIOO)^ 

Failure_A1F 
5%  2000 


^20%  Failure  A1 


D- 


Redestgn  At 
50  " 


Success_A2 
60%  VAR-(850. 1100, 13001  1100)^ 


< 


Success_B1 
90%  VAR-(1100.1400.1800|t100), 


Failure_B  1 
10%  2500 


Failure_A2 
40%  900" 


o- 


Redesign  A2 
50  “ 


< 


Success_B2 
90%  VAR'd  100.1400, 18001  1100)^ 

Failure_B2 
10%  2500  . 


Note:  In  this  hypothetical  case,  the  values  may  be  thought  of  as  $K. 

Figure  1.  Generalized  Decision  Tree  for  Risk#1  and  Two  Initial  Candidate  Risk  Response 

Actions 

We  evaluated  each  RRA  in  Figure  1  using  the  Excel  Monte  Carlo  simulation  add-in. 
Crystal  Ball.  The  selection  of  a  RRA  is  a  deterministic  event,  and  only  the  associated  outcomes 
can  be  realized.  It  would,  therefore,  be  inappropriate  to  weigh  or  combine  the  outcomes  of  the 
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two  RRAs  since  they  are  mutually  exclusive.  The  PDFs  and  risk  profiles  for  each  individual 
RRA  at  the  start  of  the  project  are  depicted  in  Figures  2a  and  2b,  respectively.  The  PDFs  are 
multimodal  and  cannot  be  represented  using  any  of  the  well-known  probability  distribution 
functions.  The  peak  for  the  “prototype”  RRA  corresponds  to  the  outcome  in  which  the 
fabrication  of  the  module  fails.  The  PDF  for  the  “direct  fabrication”  RRA  has  two  modes 
corresponding  to  the  sequence  of  events  in  which  the  first  fabrication  and  the  subsequent 
fabrication  following  redesign  both  fail. 

The  Complementary  Cumulative  Distribution  Functions  (CCDF)  or  risk  curves  are  shown 
in  Figure  2b,  in  which  the  exceedance  probability  is  the  probability  of  exceeding  a  given 
consequence  or  (1  -  the  probability  of  success).  For  example,  looking  at  the  VAR(Fab_A2) 
curve  in  Figure  2b,  one  reads  that  there  is  approximately  a  30%  probability  that  the  cost  will 
exceed  $1 ,500K.  Equivalently,  one  can  state  that  there  is  a  70%  probability  that  the  cost  will  be 
less  than  $1 ,500K.  The  risk  curve  and  the  cumulative  distribution  function  carry  identical 
information  content.  Since  we  are  focusing  on  specific  risks  and  VAR,  we  favor  the  risk  curve  or 
CCDF  because,  in  our  opinion,  it  provides  a  better  view  of  the  residual  risk  and  management 
reserve  than  the  S-curve  (or  CDF)  that  typical  represents  the  total  cost  (including  the  baseline 
and  risk  cost  elements). 

For  any  given  value  on  the  x-axis,  the  risk  curve  that  corresponds  to  the  lowest 
exceedance  probability  represents  the  lower  risk.  Figure  2b  illustrates  that  the  prototype  risk 
curve  is  significantly  lower  than  the  fabrication  risk  curve  and,  thus,  has  less  risk.  In  this 
hypothetical  but  realistic  situation,  the  investment  of  $100K  for  building  a  prototype  provides  a 
significant  return  on  the  investment  as  measured  by  the  significant  risk  reduction.  To  be  more 
precise,  the  prototype  RRA  presents  a  lower  cost  of  risk  mitigation  for  all  values  greater  than 
$200K.  For  the  manager  trying  to  decide  if  it  is  worthwhile  to  invest  in  the  prototype  option,  the 
answer  is  to  invest  as  long  as  the  anticipated  benefits  from  the  prototype  (whether  it  be  cost 
savings,  time  savings,  information,  etc.)  exceed  $200K. 


Note:  Given  the  different  scales,  the  two  PDFs  are  shown  separately  for  greater  visibility. 

Figure  2a.  Probability  Distributions  Corresponding  to  the  Two  RRA  Options  for 

Risk#1 
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Risk  #1 ,  RRA  risk  curves 


Figure  2b.  Risk  Curves  for  the  Two  RRA  Options  for  Risk  #1 

The  Dynamic  Character  of  Risk  Response  Actions 

As  a  project  progresses,  its  risk  picture  is  dynamic.  The  sources  and  consequences  of 
risks  continue  to  evolve  and  change  over  time.  As  more  information  is  obtained  about  a 
particular  risk,  the  RRA  options  might  change;  thus,  it  is  necessary  to  constantly  monitor  risk.  In 
general,  at  any  point  in  time  there  will  be  a  mix  of  acceptable  and  unacceptable  results.  The 
performance  of  the  RRAs  should  be  monitored  and  controlled  to  ensure  they  are  adequately 
mitigating  risk.  Concurrently,  management  reserves  should  be  reviewed  on  a  periodic  basis 
and  dynamically  allocated  where  needed  to  ensure  project  success.  The  Lockheed 
Management  Student  Guide  (1998,  p.  33)  states,  “Risk  management  efforts  that  fail  do  so 
because  the  risk  control  actions  did  not  keep  up  with  a  changing  program  situation.” 

As  discussed  in  the  previous  section,  we  use  risk  DTs  to  model  the  evolution  of  the 
potential  RRAs.  For  example.  Figure  3  depicts  the  Risk  #1  risk  curves  at  the  start  of 
implementation  of  the  “Prototype”  RRA  and  after  the  successful  demonstration  of  the  prototype. 
The  latter  risk  curve  moves  to  the  left  of  the  original  risk  curve  and  is  narrower,  which  reflects  a 
reduced  risk.  These  two  risk  curves  represent  the  value  of  the  unmitigated  risk  exposure  at  two 
different  points  in  time  and,  thereby,  provide  a  metric  for  the  risk  exposure  characteristics.  This 
information  is  essential  if  analysts  are  to  track  the  value  of  the  residual  exposure  versus  the 
value  or  cost  of  the  expended  RRAs  and  modify  the  RRAs  as  needed  to  ensure  mission 
success.  Note  that  if  the  risk  curve  moves  to  the  right  of  the  original  risk  curve,  it  means  that 
risk  exposure  is  increasing,  and  RRAs  need  to  be  re-evaluated. 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  EOR INEORMED  CHANGE  -  278  - 


Risk  #1 ,  Prototype  RRA 
Risk  exposure  characteristics 


0  500  1,000  1,500  2,000 


Value  at  risk,  $K 


Figure  3.  Risk  Exposure  Characteristics  for  Risk  #1  with  Development  of  a  Prototype  for 
Risk  Mitigation  at  the  Start  and  after  Successful  Demonstration 

Application  to  a  Project  with  Muitipie  Risks 

Now  consider  the  hypothetical  project  with  the  following  three  independent  risks:  Risk  #1 
depicted  in  Figure  1;  Risks  #2  and  #3  depicted  in  Figure  4a  and  4b,  respectively.  It  is  both  rich 
and  simple  enough  to  illustrate:  (1)  several  diverse  RRAs  and  their  analysis,  (2)  the  dynamic 
nature  of  the  risk  picture,  and  (3)  the  monitoring  of  individual  risks  and  allocation  of 
management  reserves.  The  approach  readily  extends  to  dependent  risks  using  different 
probability  and  outcome  values  that  reflect  causality  effects  among  the  risks. 

Figure  4a  may  be  thought  of  as  the  prime  contractor  subcontracting  the  engineering  and 
fabrication  of  a  complex  module.  The  prime  is  considering  the  following  two  options:  (1) 
subcontract  to  a  single  contractor  A,  denoted  by  the  branch  PDR_A  associated  with  the  initial 
node;  (2)  carrying  two  subcontractors  and  selecting  the  best  one  for  fabrication  at  the 
Preliminary  Design  Review  (PDR).  The  labeling  is  somewhat  cumbersome  because  each 
branch  needs  to  be  uniquely  identified.  The  PDR_A  sequence  represents  the  decision  to 
proceed  with  a  single  contractor.  The  PDR_AB  sequence  represents  the  decision  to  proceed 
with  two  contractors  and,  at  PDR,  to  select  the  best  one  for  manufacturing.  By  selecting  two 
different  contractors  with  different  offerings,  the  prime  significantly  reduces  the  probability  of 
PDR  failure.  RW  represents  the  cost  associated  with  rework;  it  is  modeled  with  a  three- 
parameter  Weibull  distribution  specified  in  terms  of  the  90*'^,  50*'^,  and  10*'^  percentiles  provided 
by  SMEs  or  historical  data. 

Figure  4b  may  be  thought  of  a  prime  contractor  who  considers  two  different  Verification 
and  Validation  (V&V)  strategies  as  a  means  for  risk  reduction.  The  branch  VVS_1_(Start  or 
CDR)  represents  the  use  of  the  standard  approach  with  planned  expenditures  of  $300K.  The 
branch  VVS_2_  (Start  or  CDR)  represents  the  use  of  a  more  thorough  V&V  strategy  with 
greater  use  of  simulation  and  planned  expenditure  of  $1 ,000K.  The  branch  RW_1_(PDR  or 
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CDR)  represents  the  rework  following  the  PDR  and  CDR,  respectively.  The  rework  is  assumed 
to  be  inversely  related  to  the  V&V  effort,  and  it  is  modeled  with  a  three-parameter  Weibull 
distribution  specified  in  terms  of  the  90‘^,  50‘^  and  10*'^  percentiles  provided  by  SMEs  or 
historical  data. 

RW  A2 


Figure  4a.  Generalized  Decision  Tree  for  Risk  #2  and  Two  Initial  Candidate  Risk 

Response  Actions 


RW  la  L 


WS_I_CDR  RW_1_COR 

300  Q  100%  W»’'(500.100Q.2000)  ^ 

VV$_2_CDR  RW_2_CDR 

Toob  Q  100%  Wei‘(  100,200.500)  ^ 

Note:  The  start  (or  PDR)  and  CDR  periods  are  shown  separately  to  simplify  the  representation  of  the 
sequence  of  events. 

Figure  4b.  Generalized  Decision  Tree  for  Risk  #3  and  Two  Initial  Candidate  Risk 

Response  Actions 
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Given  the  above  illustrative  project — with  three  risks  each  with  two  potential  RRAs — 
there  are  eight  possible  initial  Total  Project  RRAs  (TPRRA).  As  previously  discussed,  the  risk 
picture  is  dynamic  and  gets  quite  complex  as  through  time.  Consider  Risk  #1  with  development 
of  a  prototype  as  a  risk  reduction  option.  The  prototype  may  fail  or  succeed,  and  the  fabrication 
of  the  final  module  may  fail  or  succeed.  The  full  representation  of  the  set  of  all  possible 
outcomes  for  even  this  project  is  overwhelming  and  beyond  the  scope  of  a  symposium  paper. 
We,  therefore,  limit  ourselves  to  reporting  an  interesting  subset  of  the  complete  analysis  as 
follows: 

1 .  We  consider  only  two  of  the  eight  TPRRAS. 

a.  Strategy  1 .  Use  of  the  lowest  cost-mitigation  option  for  each  risk,  which  is  equivalent 
to  proceeding  as  normal — i.e.,  no  specific  RRA  for  any  of  the  three  risks.  This  is  the 
approach  that  a  risk-seeker  project  manager  would  favor. 

b.  Strategy  2.  Use  the  most  effective  RRA  for  each  risk.  This  corresponds  to:  (1 ) 
developing  a  prototype  for  Risk  #1 ,  (2)  proceeding  with  two  contractors  for  Risk  #2, 
and  (3)  implementing  a  more  thorough  V&V  effort  for  Risk  #3.  This  is  the  approach 
that  a  risk-averse  project  manager  would  favor. 

2.  For  each  strategy,  we  assume  the  best  possible  outcomes  for  the  probabilistic  nodes 
through  time  T 1 :  the  Risk  #1  prototype  and  the  Risk  #2  review  PDR_AB  succeed.  Risk 
#3  has  no  gates;  the  risk  reduction  is  directly  accounted  in  the  magnitude  of  the  rework. 
Figure  5  compares  the  initial  and  residual  risks  under  the  two  strategies. 

3.  For  each  strategy,  we  assume  the  worst  outcomes  for  the  probabilistic  nodes  through 
time  T 1 :  the  Risk  #1  prototype  and  the  Risk  #2  review  PDR_AB  fail.  Risk  #3  has  no 
gates;  the  risk  reduction  is  directly  accounted  in  the  magnitude  of  the  rework.  Figure  6 
compares  the  initial  and  residual  risks  under  the  two  strategies. 

4.  For  convenience,  we  also  report  the  50*'^,  80*'^,  and  mean  values  for  the  aggregated  risks 
and  individuals  risks  for  strategies  1  and  2  in  Tables  1  and  2,  respectively. 

Useful  Information  about  Risk 

We  now  make  a  few  brief  observations.  By  plotting  the  risk  curves  over  time  for  each 
strategy,  we  can  see  from  Figure  5  that  if  the  best  outcome  is  realized,  both  strategies  reduce 
risk  (as  seen  by  the  T1  curves  moving  left  and  becoming  more  vertical  than  the  start  curves 
over  most  of  the  range  of  analysis).  Likewise,  we  see  that  if  the  worst  outcome  prevails  as 
shown  in  Figure  6,  then  both  strategies  actually  increase  the  cost  risk  exposure  of  the  project. 
Graphing  risk  curves  over  time  thus  provides  a  metric  to  measure  the  success  of  risk  mitigation 
efforts. 
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Figure  5.  Risk  Exposure  Characteristics  for  a  Risk-seeking  Strategy  (Strategy  1)  and  a 
Risk-averse  Strategy  (Strategy  2),  Assuming  Good  Luck  Prevails  on  the  Project 


Figure  6.  Risk  Exposure  Characteristics  for  a  Risk-seeking  Strategy  (Strategy  1)  and  a 
Risk-averse  Strategy  (Strategy  2),  Assuming  Murphy’s  Law  Prevails  on  the  Project 

But  the  graphs  provide  even  more  information.  They  allow  us  to  consider  the  risk 
mitigation  qualities  of  each  strategy  and  to  quantify  our  risk  exposure.  This  information  can  be 
used  to  choose  between  the  two  strategies.  Note  that  under  the  best-case  scenario  (Figure  5), 
at  the  start  of  the  project,  Strategy  1  offers  a  lower  risk  exposure  below  $1 ,500K,  while  Strategy 
2  offers  a  lower  risk  exposure  above  that  value.  Both  strategies  are  equal  in  terms  of 
exceedence  probability  (60  %)  at  the  “breakeven”  point  of  $1 ,500K.  What  do  we  gain  by 
extending  the  analysis  to  time  T 1  ?  We  see  that  the  “breakeven”  point  is  lower  ($1 ,200K),  and 
the  risk  at  that  point  is  also  lower  (40%).  So,  which  one  is  the  best  choice?  If  we  were  optimists 
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and  certain  that  the  best  outcome  would  be  realized,  we  could  make  a  choice  based  on  the 
expected  benefits.  As  long  as  the  expected  benefits  of  the  RRA  are  greater  than  $1 ,200,  we 
would  choose  Strategy  2.  But  of  course,  we  have  no  such  assurance,  so  let’s  examine  the 
worst-case  scenario. 

Figure  6  shows  the  results  of  implementing  each  strategy  over  time  assuming  the  worst 
outcome  (Murphy’s  Law).  As  expected,  the  risk-seeking  Strategy  1  significantly  increases  our 
cost  risk  exposure  when  things  go  bad,  but  the  more  conservative  Strategy  2  is  much  less 
sensitive  to  bad  outcomes.  In  fact,  at  T1 ,  Strategy  2  dominates  Strategy  1 —  meaning  it  has  a 
lower  risk  for  any  value.  If  we  were  pessimists,  our  choice  would  be  simple:  Strategy  2  is 
especially  effective  in  providing  insurance  against  the  worst  outcomes. 

Which  strategy  is  chosen  depends  on  the  decision-maker’s  risk  aversion.  Is  he/she  an 
optimist  or  a  pessimist?  In  either  case,  if  the  expected  benefits  of  risk  mitigation  exceed 
$1,200K,  Strategy  2  is  the  best  choice.  We  believe  examining  risk  information  in  this  way 
provides  useful  insight  and  helps  project  managers  make  better  choices. 
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Table  1.  Strategy  1  Risk  Characteristics  at  Start  and  at  T1,  Assuming  that  Murphy’s  Law 

Prevails  on  the  Project 
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Table  2.  Strategy  2  Risk  Characteristics  at  Start  and  at  T1,  Assuming  that  Murphy’s  Law 

Prevails  on  the  Project 
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Tables  1  and  2  suggest  that  the  use  of  mean  values  is  not  necessarily  a  cautious 
approach  for  planning  project  contingency. 
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Conclusions 


We  have  presented  a  method  for  evaluating  and  tracking  project-specific  risks  at  the 
microscopic  level.  This  type  of  analysis,  as  opposed  to  the  macroscopic-level  risk  analysis,  is 
essential  for  risk  management.  While  the  macro  level  provides  some  information  about  total 
cost  risk,  the  micro  level  allows  the  project  manager  to  plan  and  control  risk  response  actions 
that  influence  total  cost  risk. 

We  demonstrated  the  use  of  risk  decision  trees  to  model  the  evolution  of  the  potential 
RRAs,  and  we  used  risk  curves  to  evaluate  the  risk.  We  believe  risk  curves  are  better  than  the 
expected-value  results  usually  given  by  traditional  DT  analysis  because  they  contain  all  the  risk 
information  both  in  terms  of  probabilities  and  value  at  risk.  This  thorough  approach  allows 
management  to  consider  what  they  mean  by  “acceptable”  risk  and  explicitly  models  the  tradeoff 
between  risk  and  benefits  of  any  given  RRA. 

We  recommend  the  use  of  risk  curves  to  evaluate  the  performance  of  RRA  and  to  track 
their  performance  over  time.  If  the  RRA  is  working  (at  reducing  risk),  we  should  see  the 
corresponding  risk  curve  move  to  the  left  and/or  become  more  vertical.  This  tracking  over  time 
is  key  to  understanding  the  dynamic  nature  of  risk  management  and  can  reveal  necessary 
changes  in  strategy. 

Risk  curves  derived  from  Monte  Carlo  simulation  on  DTs  are  particularly  useful  when 
analysts  are  comparing  different  risk-mitigation  strategies.  The  “breakeven”  points  help  the  risk 
manager  understand  the  conditions  under  which  each  strategy  is  most  appropriate.  Combined 
with  scenario  analysis,  it  offers  an  opportunity  to  make  cost-benefit  tradeoffs  among  strategies. 

We  think  that  these  results  provide  the  detailed  information  that  program  managers  need 
and  want  when  they  face  hard  decisions  on  programs.  There  is  a  cost  for  this  type  of  analysis, 
but  it  is  small  considering  the  potential  benefits.  The  proposed  approach  is  both  practical  and 
mathematically  valid  and  can  be  implemented  using  commercially  available  tools  such  as 
Crystal  Ball  and  @Risk.  The  challenge  is  to  start  implementing  these  more  refined  cost  models 
and  risk  management  practices. 
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2003  -  2008  Sponsored  Research  Topics 

Acquisition  Management 

■  Software  Requirements  for  OA 

■  Managing  Services  Supply  Chain 

■  Acquiring  Combat  Capability  via  Public-Private  Partnerships  (PPPs) 

■  Knowledge  Value  Added  (KVA)  +  Real  Options  (RO)  Applied  to  Shipyard 
Planning  Processes 

■  Portfolio  Optimization  via  KVA  +  RO 

■  MOSA  Contracting  Implications 

■  Strategy  for  Defense  Acquisition  Research 

■  Spiral  Development 

■  BCA:  Contractor  vs.  Organic  Growth 

Contract  Management 

■  USAF  IT  Commodity  Council 

■  Contractors  in  21st  Century  Combat  Zone 

■  Joint  Contingency  Contracting 

■  Navy  Contract  Writing  Guide 

■  Commodity  Sourcing  Strategies 

■  Past  Performance  in  Source  Selection 

■  USMC  Contingency  Contracting 

■  Transforming  DoD  Contract  Closeout 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 

Financial  Management 

■  PPPs  and  Government  Financing 

■  Energy  Saving  Contracts/DoD  Mobile  Assets 

■  Capital  Budgeting  for  DoD 

■  Financing  DoD  Budget  via  PPPs 

■  ROI  of  Information  Warfare  Systems 

■  Acquisitions  via  leasing:  MPS  case 

■  Special  Termination  Liability  in  MDAPs 
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Human  Resources 


■  Learning  Management  Systems 

■  Tuition  Assistance 

■  Retention 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 

Logistics  Management 

■  R-TOC  Aegis  Microwave  Power  Tubes 

■  Privatization-NOSL/NAWCI 

■  Army  LOG  MOD 

■  PBL  (4) 

■  Contractors  Supporting  Military  Operations 

■  RFID  (4) 

■  Strategic  Sourcing 

■  ASDS  Product  Support  Analysis 

■  Analysis  of  LAV  Depot  Maintenance 

■  DiffusionA/ariability  on  Vendor  Performance  Evaluation 

■  Optimizing  CIWS  Lifecycle  Support  (LOS) 

Program  Management 

■  Building  Collaborative  Capacity 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  Aegis  and  SSDS 

■  Business  Process  Reengineering  (BPR)  for  LOS  Mission  Module 
Acquisition 

■  Terminating  Your  Own  Program 

■  Collaborative  IT  Tools  Leveraging  Competence 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our 
website:  www.acquisitionresearch.orq 
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Dynamic  cost  risk  assessment  for 
controlling  the  cost  of  naval  vessels 

Edouard  Kujawski,  Associate  Professor,  Systems  Engineering  Department 
Diana  Angelis,  Associate  Professor,  Defense  Research  Management  Institute 
Jeffrey  Kline,  Senior  Lecturer,  Operations  Research  Department 

Naval  Postgraduate  School 
Monterey,  California 


Cost  risk  analysis  a  la  Dilbert 


THIS  BUDGET  UXKJLO  ^ 

ONLY  UmK  IF  TH£  , 
PROJECT  ENCOUNTERED 
NO  PROBLEtAS  WHAT-  I 
SOEVER 


^LL  PROJECTS  HAVE 
,  UNEXPECTED  PROBLEfAS 
THEREFORE.  THIS 
BUDGET  IS  AL.*AOST 
^^TAIf^Y  WRONG.  / 


LEADERS  DO 
NOT  PLAN  FOR 

failure 


^OO  LEADERS  fAAKE  ^ 

DECEPTIVE  FORECASTS 
ANO  LATER  ACT  SHOCKED 
WHEN  THINGS  DONT  I 

V^ORK  OUT?  / 


f: 


A  LEADER  FIRST  fAAKES 
HirsSELF  BELIEVE  THE 
LIE.  THUS  TURNING 
OEaPTION  INTO  AN 
INSPIRING  FORrA  OF 
OPTirAISTA.  OBSERVE. 


li 


THE  SWELLING  WILL 
GO  DOWN  IN  A  FEW 
HOURS  THEN  WELL 
HAVE  A  PERFECT 
BUDGET.  ^ 


S  O 
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Cost-overrun  problem**  Deja-vu 


"Their  Judgment  was  based  more  on  wishful  thinking 
than  on  sound  calculations  of  probabilities. " 

Thucydides,  431  B.C.E. 


>  Thucydides'  observation  is  very  insightful  and  still 
appropriate  today 

^  Significant  problems  in  cost  estimates  (2006  RAND  study) 

■  Systematic  bias  toward  underestimating  weapon  systems 

■  Substantial  uncertainty  in  cost  estimates 


^  Congress  has  concerns  about  shipbuilding  estimates 

Navy  Times,  March  20,  2008, "  Analyst:  Lawmakers  do  not  trust 
Navy  numbers" 
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The  use  of  Probabilistic  Cost  Analysis  (PCA) 


>  Major  shift  in  RAC)  and  complex  projects  from  deterministic 
to  probabilistic  cost  analysis 

■  Proper  framework  for  handling  cost  uncertainties 

-  Systemic  problems 

-  Project  specific  risks 


>  DoD  recognizes  that  uncertainty  or  risk  is  an  important 
aspect  of  cost  analysis 

■  NAVSEA  05C  implements  PCA  in  the  Planning, 
Programmatic,  Budgeting,  and  Execution  System 


>  Dr.  Etter,  Former  Assistant  Secretary  of  the  Navy  for 
Research,  Development,  and  Acquisition 

“Program  managers  not  only  need  to  know  a  realistic  cost 
estimate  for  their  program,  they  need  to  know  the  percent 
probability  of  achieving  that  target." 
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Representative  probabilistic  cost  analysis 


55,716  63,398  71,080  78,763  86,445  94,127  101,810  109,492  117,174  124,857 


>  How  much  risk  are  the  stakeholders  willing  to  accept? 

>  Choosing  cost  point  requires  understanding  consequences. 


BUT  is  today's  PCA  the  silver  bullet  that  slays  the  cost  overrun 
problem? 
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Budget  allocation  impacts  project  cost  and 

probability  of  success 
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Mythical  Projects 
“1007o  Rational"  project  team 

Each  project  spends  only  as 
necessary  to  satisfy  requirements 

Actual  cost  may  be  less  than 
budgeted  costs 


Real  Projects 

Human  <&  organizational  influences 

MAIMS  principle:  "Money  Allocated 
Is  Money  Spent" 

Actual  cost  increases  with  higher 
allocated  budget 


50 


>  High  cost  NEED  NOT  provide  high  probability  of  success. 

>  Choosing  cost  point  requires  understanding  consequences. 

>  Today's  typical  PCA  relies  on  expected  values  and  underestimates  cost 
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Probabilistic  cost  analysis 
A  physician  metaphor 


Adapted  from  Yacov  Y.  Haimes,  NPS  2007 
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Mrs.  Jones,  the  test  shows  that  your  current  blood  cholesterol 
level  and  weight  are  much  higher  than  the  normal  levels;  also 
given  that  you  have  been  smoking  in  the  past  20  years,  I  believe 
that  you  are  vulnerable  to  a  heart  attack. 


Oh?! 


So,  see  you  next  year.  Good  Bye. 


Risk  Assessment  Triplet  Questions 
[Kaplan  and  Garrick,  1981]; 

-  “What  can  go  wrong?” 

-  “What  is  the  likelihood?” 

-  “What  are  the  consequences?” 


Dynamic  cost  risk  management 
A  physician  metaphor 


Mrs.  Jones,  I  think  you  should  quit  smoking,  exercise,  and 
follow  a  healthy  diet.  Alternatively,  I  can  prescribe  some 
medications  for  you  now,  which  can  help  bring  down  your 
blood  cholesterol  level  more  quickly,  but  there  might  be 
some  side  effects.  My  recommendation  Is  to  do  both. 


/^lii  h 

Adapted  from  Yacov  Y.  Haimes,  NPS  2007 


Risk  Management  Triplet  Questions 
[Haimes,  1991]: 

-  “What  can  be  done  and  what 

options  are  available?” 

-  “What  are  the  tradeoffs  in  terms  of 

all  costs,  benefits,  and  risks?” 

-  “What  are  the  impacts  of  current 

decisions  on  future  options?” 
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Sources  of  cost  uncertainty 


Macroscopic  analysis 

■  Economic,  Materials  c&  Labor, 
Learning  rates 

■  Effectively  modeled  using 
classical  PDFs 

-  Triangular,  Beta,  Lognormal, 
Weibull  distributions,  ... 

©  But  these  only  constitute  a 
fraction  of  today's  typical 
project  risk  drivers 


Microscopic  analysis 

■  Project-specific,  high 
consequence  risks 

-  Technology,  Design,  Change 
orders,... 

■  Need  to  be  analyzed  within 
the  framework  of  a  complete 
risk  management  effort 

■  Modeling  and  analysis  tools 
Decision  trees,  Influence  diagrams 
Monte-Carlo  simulation 
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Integrate  PCA  Into  the  DoD 
risk  management  process 


Develop  offers  using  nultlple  assessrrents. 


Rnancal  controls  may  not  be  adequate  to  avoid  waste,  fraud 

and  abuse.  Schedule  impacts  due  to  FTA  witholding  funds.  Osst 
irrpacts  due  to  fraud,  waste,  and  abuse. 


Cost  and  schedule  impacts  due  to  scope  changes  caused  by 
cormunity/pollticat  pressure  before  the  completion  of  final 


;t  iirpacl  of  schedule  delays. 


ferform  annual  ac 


Fferf  orm  f  nancial  rranagerrent  oversi^lor 


Develop  and  irrplerrent  strict  change  order  decision  matrix  that 
requires  the  approval  of  senor  Grantee  executives  and  the  FTA  for 
major  scope  changes  after  the  conclusion  of  the  environrrental 


jr  easy  future  connectivity  tc 


A  (Use  contngency) 


It^nagemant  Action 


Ittenagement  Action 


Ittinagerrent  Action 


Ittenagement  Action 


lubnagement  Action 


Ittenagemant  Action 


Ittenagemant  Action 


Ittenagement  Action 


Risk 
Mitigation  Plan 
Implementation 
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Extended  PCA 

Dynamic  cost  risk  management 

■  Identify  project-specific  risks 

■  Screen  for  further  analysis  and  mitigation 

■  bevelop  risk  response  actions 

■  Model  risk  and  actions  using  decision  tree 

■  Model  cost  uncertainty  with  continuous  and  discrete 
distributions 

■  Use  simulation  to  explore  value  of  risk  response  actions 
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Total  Project  Cost  = 
Baseline  Cost  +  Risk  Costs 


©  The  microscopic  view  is  a  powerful  risk  analysis  method 
©  However,  it  is  too  cumbersome  to  individually  analyze  every  risk 
^  It  complements  and  needs  to  be  integrated  within  the  PCA 
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Project  cost  and  uncertainty  over  twe 
A  very  dynan^lc  picture 
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Example  #1  -  Single  risk.  Risk  #1 
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Enhanced  Decision  Tree  Analysis  (DTA) 


Standard  DTA 

■  Information  lost  in  folding 
back 

■  Decisions  based  on  expected 
value 

■  Representation  quickly 
becomes  too  bushy 


Enhanced  DTA 

■  Spectrum  of  outcomes 
explicit 

■  Decisions  based  on  risk 
profiles  and  DM's  attitude 
toward  risk 

■  Compact  representation 

■  Readily  implemented  using 
commercially  available  SW 


>  Enhanced  DTA  provides  more  complete  and  meaningful 
information  than  standard  DTA  for  the  same  data. 
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Multiple  risks 


^  Consider  a  project/program  with  m  base  cost  elements  {BCj}  and  n 
credible,  high-consequence  risks  and/or  opportunities  {R,} 

-  Each  base  cost  element  may  be  modeled  as  a  point  estimate  or 
continuous  PDF 

-  Each  risk  is  characterized  by  a  probability  of  occurrence  pi  a  spectrum 
of  possible  outcomes  with  a  PDF  Lj(x) 

^  Think  of  project/program  of  risks  is  a  risk/opportunity  portfolio 
with  a  generalized  discrete  PDF 


Rs  W  =  |(Pi .  L,  (x)) ,  (pj ,  Lj  (x)) , . . . ,  (p„ ,  L„  (x)) 


^  The  total  cost  Is  then  the  probabilistic  sum  of  the  m  base  cost 
elements  and  n  risk-driver  costs 


m 


TCix)  =  ^  SC.(x)©^  piLi{x) 


i  =  l 


i  =  l 
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Example  #  2  -  Project  with  3  risks 


Risk  #1  -  slide  14 
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Risk  #2:  high-risk  subcontract 

-  Single  contractor 

-  Two  contractors,  downselect  at  PDR 


Risk  #3 


RW  la  L 
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Risk  #3:  contractor  considers  two 
VAV  strategies  for  risk  reduction 

-  Standard  VAV 

-  Extensive  modeling  and  simulation 


Rool  R3  CDR 
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Example  #  2 -Dynamic  risk  curves 
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Example  #  2  -  Range  of  Risk 
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Example  #2  -  Comparing  Strategies 

(Best  Scenario) 
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Example  #2  -  Comparing  Strategies 

(Worst  Scenario) 
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Conclusions 


>  Project-specif  ic  cost  risks  can  be  modeled  using  decision 
trees  and  simulation 

-  Micro  level  essential  for  risk  management 

>  bynamic  risk  curve  analysis  can  be  used  to  select  and  track 
performance  of  risk  response  actions  over  time 

-  Enable  cost-risk  tradeoffs 

•  Dynamic  risk  analysis  and  management  are  key  to  improve 
project/program  technical  performance,  schedule,  and  cost 
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Further  Research 


□  Incorporate  multiple  risks  into  dynamic  risk  analysis  and 
management 

-  Solve  for  optimal  risk  mitigation  strategy 

□  Expand  results  to  budget  implications  of  cost  estimates 

□  Integrate  with  schedule  and  technical  performance 
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